Experience Optimizer Metric Descriptions

Algonomy’s automated Experience Optimizer process dynamically chooses strategies based on how they score for a specific optimization or metric.

Instead of trying to identify the value of specific strategies, Algonomy focuses on identifying the best optimization metric for determining which strategies to display. For this purpose, Algonomy’s Recommend includes various optimization metrics. We can A/B test these against Clickthrough Rate, which is default, to determine if an alternative strategy selection logic produces more incremental revenue for the specific retailer.

We strongly recommend not to change the site’s settings without conducting an A/B test to ascertain impact. Since we typically see that new metrics impact RPV by up to 1%, you want to make sure it is doing so in the positive direction before exposing it to full traffic. Metric configurations are not presently client-facing. If you want to adjust or evaluate them (via A/B), contact Algonomy Support team or Account Manager.

How It Works

By default, this metric is clickthrough rate—meaning that the engine commonly displays the strategies that demonstrate the highest engagement for the specific context, such as page, placement and category combination. It is found that this is an effective metric because it promotes product discovery.

While engagement can be important, retailers are most interested in displaying strategies that generate the most Revenue Per Visit lift, not necessarily those that get the most clicks. The challenge is that, given the number of strategies in play, it is difficult to determine a specific strategy’s contribution to a retailer’s incremental revenue. It requires very elongated A/B experimentation to reach results with any reasonable degree of confidence—and that risks exposing customers to sub-optimal strategies for prolonged periods of time.

Managing Your Metric

Within the Personalization dashboard, recommendations on all pages can be configured to use the same metric or you can assign separate ones for the Item, Add to Cart and Cart pages. This flexibility allows us to respect the diverse merchandising objectives at key points in the shopping funnel; while product discovery may be important on pages with broader context (For example, Home), deeper in the funnel, excessive clicks can compromise Conversion—and so you want to focus on recommendations safely that grow AOV.

We strongly recommend not to change the site’s settings without conducting an A/B test to ascertain impact. To change your site's metric, contact Algonomy Account Manager.

Metric Definitions

The following are the available metrics, their definitions, and underlying logic. While some are very similar, their nuances and slight computational differences result in a different strategy ranking.

Note: An “attributable click” refers to a user click on an {rr} recommendation that led to a purchase of that recommended product. Similarly, “attributable revenue” refers to the revenue from an item clicked on in {rr} recommendations.

Clickthrough Rate

The number of times that recs provided by the strategy were clicked on divided by number of times that strategy was used to show recs.

This default metric prioritizes product discovery—exposing customers to as much of the retailer’s catalog as possible—by selecting strategies that have the highest probability of being clicked for that specific context (For example, placement and category combination). It hypothesizes that maximizing recs engagement delivers the most value.

Revenue on Attributable Clicks

For each visit, split the revenue generated during that visit equally among all attributable clicks in the visit. Total this up across visit and divide by the number of times the strategy played.

Note: A visit with no attributable clicks contributes no revenue.

This metric seeks to credit strategies for customer spend beyond what’s directly attributable to recommendations. The premise is that recommendations not only get customers to buy what was recommended, but that they can have a “halo effect” that can influence the purchase of non-recommended items.

By definition, even if a visit results in a conversion, if there are no recs-attributable items in that sale, all viewed strategies are penalized. This is tantamount to saying that a strategy’s influence is only meaningful if it can get people to purchase at least one recommended item. Otherwise, any of the items purchased were probably not influenced by the presence of recommendations. Also, with this approach, if a session results in zero revenue, any viewed strategies are penalized.

Attributable Revenue

Total attributable revenue generated by the strategy divided by the number of times the strategy was shown.

This metric hypothesizes that incremental revenue is a function of recs sales and therefore maximizes the display of strategies that have the highest attributables yield per view. That is, the focus is on recommendation sales irrespective of that click’s overall impact on session revenue.

Attributable Clickthrough Rate

Total number of attributable clicks divided by the number of times the strategy was shown. This is similar to Clickthrough Rate, except that it only counts clicks that led to attributable purchases.

This metric prioritizes product discovery, but only in the context of producing recommendations revenue. Whereas CTR elevates strategies that purely have high engagement. Attributable CTR does so for ones that led to attributable purchases. In many cases, this could lead to the same strategy ranking that CTR produces, as Attributable Clicks is largely a function of Clicks.

Revenue Split Between All Clicks

For each visit, the revenue is spread equally across strategies that had clicks. Total this revenue across all visits divided by the number of times the strategy was shown.

Note: A visit with no clicks generates no revenue.

This metric suggests that a visit's revenue (entire revenue—not just recs sales), or lack thereof, can be attributed to the strategies that were clicked during that session. It effectively promotes strategies whose engagement is correlated with the most session revenue. This metric is likely the best proxy for Revenue Per Visit though it is by no means a perfect match.

Visit Length Split Between Clicks

For each visit, spread the visit length (in page views) equally across all clicks. Total these visit lengths across visits and divide by the number of times the strategy was shown.

This metric provides a correlation between site stickiness and value. As such, it seeks to promote strategies whose engagement (that is clicks) is associated with the greatest number of page views.

This is most useful for content sites and/or ones with advertising revenue streams.

Visit Length Split Between Views

For each visit, spread the visit length (in page views) equally across all strategy views. Total these visit lengths across visits and divide by the number of times the strategy was shown.

This metric provides a correlation between site stickiness and value. As such, it seeks to promote strategies whose views are associated with the greatest session lengths.

This is most useful for content sites and/or ones with advertising revenue streams.

AOV Split Between Clicks

For each purchasing visit, spread the total revenue equally across all strategies that had clicks. Total this across all visits and divide by the number of times the strategy was shown within purchasing visits in which the strategy was clicked.

This metric is similar to Revenue Split Between All Clicks except that it only takes into account converting visits. In some cases, this can be problematic because if clicking on a particular strategy is correlated to non-purchasing visits (For example, a strategy that tends to recommend very expensive items), this metric does not penalize that strategy.

AOV Split Between Attributable Clicks

For each purchasing visit, spread the total revenue equally across all strategies with attributable clicks. Total this across all visits and divide by the number of times the strategy was shown within purchasing visits in which the strategy was clicked.

This metric is similar to Revenue on Attributable Clicks except that it only takes into account converting visits. In some cases, this can be problematic because if clicking on a particular strategy is correlated to non-purchasing visits (For example, a strategy that tends to recommend very expensive items), this metric does not penalize that strategy.